Video Annotation Research Articles

Functional Analysis of English Carriers and Related Resources of Cultural Communication in Internet Media

Abstract In the Internet intelligent teaching platform, students’ demand for English cultural content is increasingly obvious. To help students quickly locate the overall content of resources in online autonomous learning, this study constructs a video annotation model for online teaching. This method classifies text by designing an optimized Bidirectional Encoder Representation from the Transformers model and designs a Text Rank keyword extraction model that integrates external knowledge and semantic feature weights. The extraction of knowledge points contained in audio and video resources can be realized. In the experimental data set, a relatively complete video content summary could be obtained by combining the first three sentences with the last two sentences. The F1 value of the classification model was up to 91.3%. In addition, the BERT-T model proposed in this article had the best effect on the experiment. Compared with the original BERT model, the macro-F1 was 0.8% higher and 0.5% higher than the Ro BERTA model. In the keyword extraction experiment, B-Text Rank was 2.19 and 2.85% higher than the traditional Text Rank in the two datasets. The experiment shows that the BERT-Text Rank network resource annotation model has excellent application performance in English online autonomous teaching and could guide students to learn.

On-the-fly point annotation for fast medical video labeling.

In medical research, deep learning models rely on high-quality annotated data, a process often laborious and time-consuming. This is particularly true for detection tasks where bounding box annotations are required. The need to adjust two corners makes the process inherently frame-by-frame. Given the scarcity of experts' time, efficient annotation methods suitable for clinicians are needed. We propose an on-the-fly method for live video annotation to enhance the annotation efficiency. In this approach, a continuous single-point annotation is maintained by keeping the cursor on the object in a live video, mitigating the need for tedious pausing and repetitive navigation inherent in traditional annotation methods. This novel annotation paradigm inherits the point annotation's ability to generate pseudo-labels using a point-to-box teacher model. We empirically evaluate this approach by developing a dataset and comparing on-the-fly annotation time against traditional annotation method. Using our method, annotation speed was faster than the traditional annotation technique. We achieved a mean improvement of AP@50 over conventional method at equivalent annotation budgets on the developed dataset. Without bells and whistles, our approach offers a significant speed-up in annotation tasks. It can be easily implemented on any annotation platform to accelerate the integration of deep learning in video-based medical research.

Reconstructive network under contrastive graph rewards for video summarization

Video summarization aims to condense video content by extracting pivotal frames or shots. Most existing methods focus on maximizing the intersection between predicted summary and ground truth, overlooking whether users can infer the content of the original video from the summary. Additionally, these approaches heavily rely on annotated data, posing limitations.Therefore, we propose a reconstructive network under contrastive graph rewards for video summarization, comprising a summary generator and a video reconstructor. The summary generator employs graph contrastive learning to distill essential video information to generate summary. Meanwhile, the video reconstructor employs reinforcement learning within an unsupervised training framework to optimize the summary generator, addressing the shortage of annotated video data in summarization tasks. Leveraging reconstruction loss, our approach ensures that predicted summary encapsulate main video content and inter-shots dependencies. Notably, we innovatively devise a mutual information maximization reconstruction reward function to preserve shared information between the summary and the original video, facilitating users in comprehending the original video content. We conduct massive experiments on the TVSum and SumMe datasets, and our network achieved F1 scores of 58.8% and 48.0%, respectively. Experimental results validate the superiority of our method over both state-of-the-art unsupervised and many supervised video summarization techniques.

Introducing surgical intelligence in gynecology: Automated identification of key steps in hysterectomy.

The analysis of surgical videos using artificial intelligence holds great promise for the future of surgery by facilitating the development of surgical best practices, identifying key pitfalls, enhancing situational awareness, and disseminating that information via real-time, intraoperative decision-making. The objective of the present study was to examine the feasibility and accuracy of a novel computer vision algorithm for hysterectomy surgical step identification. This was a retrospective study conducted on surgical videos of laparoscopic hysterectomies performed in 277 patients in five medical centers. We used a surgical intelligence platform (Theator Inc.) that employs advanced computer vision and AI technology to automatically capture video data during surgery, deidentify, and upload procedures to a secure cloud infrastructure. Videos were manually annotated with sequential steps of surgery by a team of annotation specialists. Subsequently, a computer vision system was trained to perform automated step detection in hysterectomy. Analyzing automated video annotations in comparison to manual human annotations was used to determine accuracy. The mean duration of the videos was 103 ± 43 min. Accuracy between AI-based predictions and manual human annotations was 93.1% on average. Accuracy was highest for the dissection and mobilization step (96.9%) and lowest for the adhesiolysis step (70.3%). The results of the present study demonstrate that a novel AI-based model achieves high accuracy for automated steps identification in hysterectomy. This lays the foundations for the next phase of AI, focused on real-time clinical decision support and prediction of outcome measures, to optimize surgeon workflow and elevate patient care.

Joint learning of video scene detection and annotation via multi-modal adaptive context network

The tasks of scene detection and annotation have gained impressive attention for understanding video content. The main challenges lie in mitigating the error propagation of shot detection, recognizing cuts and gradual transitions, fusing hierarchical multi-modal cues, and solving these two tasks simultaneously. To address these challenges, we propose the Multi-modal Adaptive Context Network (MACN) to jointly learn scene detection and annotation from a window partitioning perspective. As a shared task-agnostic part, we perform Window-based Cross-modal Representation (WCR) to distill complex semantic correlations from multi-modal sources for each window. Considering the long-term temporal dependency of variable-length scenes, we further develop Adaptive Context-aware Representation (ACR) to improve the performance for specific tasks. Different from previous works, scene detection is formulated as locating the starting window and its associated location offset and transition duration. Meanwhile, we assemble two multi-label sub-classifiers in different levels to predict the labels for each scene candidate. Experimental comparisons to state-of-the-art algorithms on the TAVS and ClipShots indicate that the proposed method yields promising performance in both tasks. Our code and test sample videos are released at MACN.

Exploiting Auxiliary Caption for Video Grounding

Video grounding aims to locate a moment of interest matching the given query sentence from an untrimmed video. Previous works ignore the sparsity dilemma in video annotations, which fails to provide the context information between potential events and query sentences in the dataset. In this paper, we contend that exploiting easily available captions which describe general actions, i.e., auxiliary captions defined in our paper, will significantly boost the performance. To this end, we propose an Auxiliary Caption Network (ACNet) for video grounding. Specifically, we first introduce dense video captioning to generate dense captions and then obtain auxiliary captions by Non-Auxiliary Caption Suppression (NACS). To capture the potential information in auxiliary captions, we propose Caption Guided Attention (CGA) project the semantic relations between auxiliary captions and query sentences into temporal space and fuse them into visual representations. Considering the gap between auxiliary captions and ground truth, we propose Asymmetric Cross-modal Contrastive Learning (ACCL) for constructing more negative pairs to maximize cross-modal mutual information. Extensive experiments on three public datasets (i.e., ActivityNet Captions, TACoS and ActivityNet-CG) demonstrate that our method significantly outperforms state-of-the-art methods.

Knowledge Guided Semi-supervised Learning for Quality Assessment of User Generated Videos

Perceptual quality assessment of user generated content (UGC) videos is challenging due to the requirement of large scale human annotated videos for training. In this work, we address this challenge by first designing a self-supervised Spatio-Temporal Visual Quality Representation Learning (ST-VQRL) framework to generate robust quality aware features for videos. Then, we propose a dual-model based Semi Supervised Learning (SSL) method specifically designed for the Video Quality Assessment (SSL-VQA) task, through a novel knowledge transfer of quality predictions between the two models. Our SSL-VQA method uses the ST-VQRL backbone to produce robust performances across various VQA datasets including cross-database settings, despite being learned with limited human annotated videos. Our model improves the state-of-the-art performance when trained only with limited data by around 10%, and by around 15% when unlabelled data is also used in SSL. Source codes and checkpoints are available at https://github.com/Shankhanil006/SSL-VQA.

3D visualization systems improve operator efficiency during difficult laparoscopic cholecystectomy: a retrospective blinded review of surgical videos.

3D visualization systems in laparoscopic surgery have been proposed to improve manual task handling compared to 2D, however, few studies have compared the intra-operative efficacy in laparoscopic cholecystectomy (LC). The aim of this study is to determine if there is a benefit in intra-operative efficiency when using a 3D visualization system in difficult LC compared to traditional 2D visualization systems. Retrospective analysis of 'difficult' LCs (Grades 3 or 4) was completed. The assessor was blinded as all cases were recorded and viewed in 2D only. Variables collected included time to complete steps, missed hook diathermy attempts, failed grasp attempts, missed clip attempts and preparation steps for intra-operative cholangiogram (IOC). Multiple linear regression was undertaken for time variables, Poisson regression or negative binomial regression was completed for continuous variables. Fifty-two operative videos of 'difficult' LC were reviewed. 3D systems were associated with reduced operative times, although this was not statistically significant (CI: -2.93-14.93, P-value = 0.183). Dissection of the anterior fold to achieve the critical view of safety was significantly faster by 3.55 min (CI: 1.215-9.206, P-value = 0.002), and with considerably fewer errors when using 3D systems. Fewer IOC preparation errors were observed with a 3D system compared with a 2D system. 3D systems appear to enhance operator efficiency, allowing faster completion of critical steps with fewer errors. This pilot study underscores the utility of video annotation for intra-operative assessment and suggests that, in larger multi-centre studies, 3D systems may demonstrate superior intra-operative efficiency over 2D systems during a 'difficult' LC.

Automated surgical step recognition in transurethral bladder tumor resection using artificial intelligence: transfer learning across surgical modalities.

Automated surgical step recognition (SSR) using AI has been a catalyst in the "digitization" of surgery. However, progress has been limited to laparoscopy, with relatively few SSR tools in endoscopic surgery. This study aimed to create a SSR model for transurethral resection of bladder tumors (TURBT), leveraging a novel application of transfer learning to reduce video dataset requirements. Retrospective surgical videos of TURBT were manually annotated with the following steps of surgery: primary endoscopic evaluation, resection of bladder tumor, and surface coagulation. Manually annotated videos were then utilized to train a novel AI computer vision algorithm to perform automated video annotation of TURBT surgical video, utilizing a transfer-learning technique to pre-train on laparoscopic procedures. Accuracy of AI SSR was determined by comparison to human annotations as the reference standard. A total of 300 full-length TURBT videos (median 23.96 min; IQR 14.13-41.31 min) were manually annotated with sequential steps of surgery. One hundred and seventy-nine videos served as a training dataset for algorithm development, 44 for internal validation, and 77 as a separate test cohort for evaluating algorithm accuracy. Overall accuracy of AI video analysis was 89.6%. Model accuracy was highest for the primary endoscopic evaluation step (98.2%) and lowest for the surface coagulation step (82.7%). We developed a fully automated computer vision algorithm for high-accuracy annotation of TURBT surgical videos. This represents the first application of transfer-learning from laparoscopy-based computer vision models into surgical endoscopy, demonstrating the promise of this approach in adapting to new procedure types.

Real-World Balance Assessment While Standing for Fall Prediction in Older Adults.

Postural control naturally declines with age, leading to an increased risk of falling. Within clinical settings, the deployment of balance assessments has become commonplace, facilitating the identification of postural instability and targeted interventions to forestall falls among older adults. Some studies have ventured beyond the controlled laboratory, leaving, however, a gap in our understanding of balance in real-world scenarios. Previously reported algorithms were used to build a finite-state machine (FSM) with four states: walking, turning, sitting, and standing. The FSM was validated against video annotations (gold standard) in an independent dataset with data collected on 20 older adults. Later, the FSM was applied to data from 168 community-dwelling older people in the InCHIANTI cohort who were evaluated both in the laboratory and then remotely in real-world conditions for a week. A 70/30 data split with recursive feature selection and resampling techniques was used to train and test four machine-learning models. In identifying fallers, duration, distance, and mean frequency computed during standing in real-world settings revealed significant relationships with fall risk. Also, the best-performing model (Lasso Regression) built on real-world balance features had a higher area under the curve (AUC, 0.76) than one built on lab-based assessments (0.57). Real-world balance features differ considerably from laboratory balance assessments (Romberg test) and have a higher predictive capacity for identifying patients at high risk of falling. These findings highlight the need to move beyond traditional laboratory-based balance measures and develop more sensitive and accurate methods for predicting falls.

Man and the Machine: Effects of AI-assisted Human Labeling on Interactive Annotation of Real-Time Video Streams

AI-assisted interactive annotation is a powerful way to facilitate data annotation – a prerequisite for constructing robust AI models. While AI-assisted interactive annotation has been extensively studied in static settings, less is known about its usage in dynamic scenarios where the annotators operate under time and cognitive constraints, e.g., while detecting suspicious or dangerous activities from real-time surveillance feeds. Understanding how AI can assist annotators in these tasks and facilitate consistent annotation is paramount to ensure high performance for AI models trained on these data. We address this gap in interactive machine learning (IML) research, contributing an extensive investigation of the benefits, limitations, and challenges of AI-assisted annotation in dynamic application use cases. We address both the effects of AI on annotators and the effects of (AI) annotations on the performance of AI models trained on annotated data in real-time video annotations. We conduct extensive experiments that compare annotation performance at two annotator levels (expert and non-expert) and two interactive labelling techniques (with and without AI-assistance). In a controlled study with N = 34 annotators and a follow up study with 51963 images and their annotation labels being input to the AI model, we demonstrate that the benefits of AI-assisted models are greatest for non-expert users and for cases where targets are only partially or briefly visible. The expert users tend to outperform or achieve similar performance as AI model. Labels combining AI and expert annotations result in the best overall performance as the AI reduces overflow and latency in the expert annotations. We derive guidelines for the use of AI-assisted human annotation in real-time dynamic use cases.

How to track and segment fish without human annotations: a self-supervised deep learning approach

Tracking fish movements and sizes of fish is crucial to understanding their ecology and behaviour. Knowing where fish migrate, how they interact with their environment, and how their size affects their behaviour can help ecologists develop more effective conservation and management strategies to protect fish populations and their habitats. Deep learning is a promising tool to analyse fish ecology from underwater videos. However, training deep neural networks (DNNs) for fish tracking and segmentation requires high-quality labels, which are expensive to obtain. We propose an alternative unsupervised approach that relies on spatial and temporal variations in video data to generate noisy pseudo-ground-truth labels. We train a multi-task DNN using these pseudo-labels. Our framework consists of three stages: (1) an optical flow model generates the pseudo-labels using spatial and temporal consistency between frames, (2) a self-supervised model refines the pseudo-labels incrementally, and (3) a segmentation network uses the refined labels for training. Consequently, we perform extensive experiments to validate our method on three public underwater video datasets and demonstrate its effectiveness for video annotation and segmentation. We also evaluate its robustness to different imaging conditions and discuss its limitations.

Building your own mountain: the effects, limits, and drawbacks of cold-water coral ecosystem engineering

Abstract. Framework-forming cold-water corals (CWCs) are ecosystem engineers that build mounds in the deep sea that can be up to several hundred metres high. The effect of the presence of cold-water coral mounds on their surroundings is typically difficult to separate from environmental factors that are not affected by the mounds. We investigated the environmental control on and the importance of ecosystem engineering for cold-water coral reefs using annotated video transect data, spatial variables (MEMs), and hydrodynamic model outputs in a redundancy analysis and with variance partitioning. Using available hydrodynamic simulations with cold-water coral mounds and simulations where the mounds were artificially removed, we investigated the effect of coral mound ecosystem engineering on the spatial configuration of reef habitat and discriminated which environmental factors are and which are not affected by the mounds. We find that downward velocities in winter, related to non-engineered environmental factors, e.g. deep winter mixing and dense-water cascading, cause substantial differences in reef cover at the broadest spatial scale (20–30 km). Such hydrodynamic processes that stimulate the food supply towards the corals in winter seem more important for the reefs than cold-water coral mound engineering or similar hydrodynamic processes in summer. While the ecosystem-engineering effect of cold-water corals is frequently discussed, our results also highlight the importance of non-engineered environmental processes. We further find that, due to the interaction between the coral mound and the water flow, different hydrodynamic zones are found on coral mounds that likely determine the typical benthic zonations of coral rubble at the mound foot, the dead coral framework on the mound flanks, and the living corals near the summit. Moreover, we suggest that a so-called Massenerhebung effect (well known for terrestrial mountains) exists, meaning that benthic zonation depends on the location of the mound rather than on the height above the seafloor or water depth. Our finding that ecosystem engineering determines the configuration of benthic habitats on cold-water coral mounds implies that cold-water corals cannot grow at deeper depths on the mounds to avoid the adverse effects of climate change.

Freezing of gait assessment with inertial measurement units and deep learning: effect of tasks, medication states, and stops

BackgroundFreezing of gait (FOG) is an episodic and highly disabling symptom of Parkinson’s Disease (PD). Traditionally, FOG assessment relies on time-consuming visual inspection of camera footage. Therefore, previous studies have proposed portable and automated solutions to annotate FOG. However, automated FOG assessment is challenging due to gait variability caused by medication effects and varying FOG-provoking tasks. Moreover, whether automated approaches can differentiate FOG from typical everyday movements, such as volitional stops, remains to be determined. To address these questions, we evaluated an automated FOG assessment model with deep learning (DL) based on inertial measurement units (IMUs). We assessed its performance trained on all standardized FOG-provoking tasks and medication states, as well as on specific tasks and medication states. Furthermore, we examined the effect of adding stopping periods on FOG detection performance.MethodsTwelve PD patients with self-reported FOG (mean age 69.33 ± 6.02 years) completed a FOG-provoking protocol, including timed-up-and-go and 360-degree turning-in-place tasks in On/Off dopaminergic medication states with/without volitional stopping. IMUs were attached to the pelvis and both sides of the tibia and talus. A temporal convolutional network (TCN) was used to detect FOG episodes. FOG severity was quantified by the percentage of time frozen (%TF) and the number of freezing episodes (#FOG). The agreement between the model-generated outcomes and the gold standard experts’ video annotation was assessed by the intra-class correlation coefficient (ICC).ResultsFor FOG assessment in trials without stopping, the agreement of our model was strong (ICC (%TF) = 0.92 [0.68, 0.98]; ICC(#FOG) = 0.95 [0.72, 0.99]). Models trained on a specific FOG-provoking task could not generalize to unseen tasks, while models trained on a specific medication state could generalize to unseen states. For assessment in trials with stopping, the agreement of our model was moderately strong (ICC (%TF) = 0.95 [0.73, 0.99]; ICC (#FOG) = 0.79 [0.46, 0.94]), but only when stopping was included in the training data.ConclusionA TCN trained on IMU signals allows valid FOG assessment in trials with/without stops containing different medication states and FOG-provoking tasks. These results are encouraging and enable future work investigating automated FOG assessment during everyday life.

Automated Identification of Key Steps in Robotic-Assisted Radical Prostatectomy Using Artificial Intelligence.

The widespread use of minimally invasive surgery generates vast amounts of potentially useful data in the form of surgical video. However, raw video footage is often unstructured and unlabeled, thereby limiting its use. We developed a novel computer-vision algorithm for automated identification and labeling of surgical steps during robotic-assisted radical prostatectomy (RARP). Surgical videos from RARP were manually annotated by a team of image annotators under the supervision of 2 urologic oncologists. Full-length surgical videos were labeled to identify all steps of surgery. These manually annotated videos were then utilized to train a computer vision algorithm to perform automated video annotation of RARP surgical video. Accuracy of automated video annotation was determined by comparing to manual human annotations as the reference standard. A total of 474 full-length RARP videos (median 149 minutes; IQR 81 minutes) were manually annotated with surgical steps. Of these, 292 cases served as a training dataset for algorithm development, 69 cases were used for internal validation, and 113 were used as a separate testing cohort for evaluating algorithm accuracy. Concordance between artificial intelligence‒enabled automated video analysis and manual human video annotation was 92.8%. Algorithm accuracy was highest for the vesicourethral anastomosis step (97.3%) and lowest for the final inspection and extraction step (76.8%). We developed a fully automated artificial intelligence tool for annotation of RARP surgical video. Automated surgical video analysis has immediate practical applications in surgeon video review, surgical training and education, quality and safety benchmarking, medical billing and documentation, and operating room logistics.

Implementation of Artificial Intelligence-Based Computer Vision Model for Sleeve Gastrectomy: Experience in One Tertiary Center.

Sleeve gastrectomy (SG) is the most common metabolic and bariatric procedure performed. Leveraging artificial intelligence (AI) for automated real-time data structuring and annotations of surgical videos has immense potential of clinical applications. This study presents initial real-world implementation of AI-based computer vision model in sleeve gastrectomy (SG) and external validation of accuracy of safety milestone annotations. A retrospective single-center study of 49 consecutive SG videos was captured and analyzed by the AI platform (December 2020-August 2023). A bariatric surgeon viewed all videos and assessed safety milestones adherence, compared to the AI annotations. Patients' data were retrieved from the bariatric unit registry. SG total duration was 47.5min (interquartile range 36-64). Main steps included preparation (12.2%), dissection of the greater curvature (30.8%), gastric transection (28.5%), specimen extraction (7.2%), and final inspection (14.4%). Out of body time comprised 6.9% of the total video. Safety milestones components and AI-surgeon agreements included the following: bougie insertion (100%), distance from pylorus ≥ 2cm (100%), parallel to lesser curvature (98%), fundus mobilization (100%), and distance from esophagus ≥ 1cm (true-100%, false-13.6%; kappa coefficient 0.2, p = 0.006). Intraoperative complications included notable hemorrhage (n = 4) and parenchymal injury (n = 1). The AI model provides a fully automated SG video analysis. Outcomes suggest its accuracy in four of five safety milestone annotations. This data is valuable, as it reflects objective performance measures which can help us improve the surgical quality and efficiency of SG. Larger cohorts will enable SG standardization and clinical correlations with outcomes, aiming to improve patients' safety.

Can't read my broker face?—Tracing a motif and metaphor of expert knowledge through audiovisual images of the financial crisis

AbstractBased on the question of the representability of economy and economics in audiovisual media, developments on the financial markets have often been discussed as a depiction problem. The abstractness and complexity of economic interrelations seem to defy classical modes of storytelling and dramatization. Nevertheless, public opinion about economic changes and dependencies crucially relies on audiovisual media. But how can the public communicate in images, sounds, and words about forces that are out of sight and out of reach, and can supposedly only be adequately grasped by experts? In a case study on audiovisual images of the global financial crisis (2007–), this paper tracks and analyzes a recurring motif: the staging of expert knowledge as close‐ups of expressive faces vis‐à‐vis computer screens in television news, documentaries, as well as feature films. It draws on the use of digital tools for corpus exploration (reverse image search) and the visualization of video annotations. By relating and comparing different staging strategies by which these “broker faces” become embodiments of turbulent market dynamics, the paper proposes to not regard them as repeated instantiations of the same metaphor, but as a developing web of cinematic metaphors. Different perspectives (news of market developments or historical accounts of crisis developments) and affective stances toward the global financial crisis are expressed in these variations of the face‐screen constellation. The paper thus presents a selection of different appearances of “broker faces” as a medium for an audiovisual discourse of the global financial crisis. A concluding analysis of a scene from Margin Call focuses on its specific intertwining of expert and screen as an ambivalent movement figuration of staging insight. Between the feeling of discovery (of a potential future threat) and the sense of being haunted (by a menacing force), the film stages the emergence of a “broker face” in an atmospheric tension between suspense and melancholy. We argue that the film thereby reframes the motif and poses questions of agency, temporality, and expert knowledge.

‘Sewing together’: An idiographic study of a music therapist’s experiences of a group at a special school for children and young people with severe and complex special educational needs

This research was commissioned by a school in London which provides for children and young people aged 2 to 16 with a wide range of learning difficulties. This single-participant study uses video annotation, video elicitation interview and interpretative phenomenological analysis (IPA) to explore perceptions of processes in a music therapy group, from the point of view of a music therapist. Two tables of annotations present a description of events in each video clip in a timeline. The IPA interview analysis produced 25 emergent themes, grouped into four superordinate themes (‘Balancing diverse needs’; ‘Spontaneity’; ‘Therapists’ musical stance’; ‘Group dynamics’). Parallels are drawn with other music therapy groups described in the literature and with Foulkes’ group analytic psychotherapy. The study elucidates an approach to group music therapy which provides a space for spontaneous and meaningful interaction, both through music and other modalities.

Comparison of manual, machine learning, and hybrid methods for video annotation to extract parental care data

Measuring parental care behaviour in the wild is central to the study of animal ecology and evolution, but it is often labour‐ and time‐intensive. Efficient open‐source tools have recently emerged that allow animal behaviour to be quantified from videos using machine learning and computer vision techniques, but there is limited appraisal of how these tools perform compared to traditional methods. To gain insight into how different methods perform in extracting data from videos taken in the field, we compared estimates of the parental provisioning rate of wild house sparrows Passer domesticus from video recordings. We compared four methods: manual annotation by experts, crowd‐sourcing, automatic detection based on the open‐source software DeepMeerkat, and a hybrid annotation method. We found that the data collected by the automatic method correlated with expert annotation (r = 0.62) and further show that these data are biologically meaningful as they predict brood survival. However, the automatic method produced largely biased estimates due to the detection of non‐visitation events, while the crowd‐sourcing and hybrid annotation produced estimates that are equivalent to expert annotation. The hybrid annotation method takes approximately 20% of annotation time compared to manual annotation, making it a more cost‐effective way to collect data from videos. We provide a successful case study of how different approaches can be adopted and evaluated with a pre‐existing dataset, to make informed decisions on the best way to process video datasets. If pre‐existing frameworks produce biased estimates, we encourage researchers to adopt a hybrid approach of first using machine learning frameworks to preprocess videos, and then to do manual annotation to save annotation time. As open‐source machine learning tools are becoming more accessible, we encourage biologists to make use of these tools to cut annotation time but still get equally accurate results without the need to develop novel algorithms from scratch.

Analyzing Surgical Technique in Diverse Open Surgical Videos With Multitask Machine Learning

To overcome limitations of open surgery artificial intelligence (AI) models by curating the largest collection of annotated videos and to leverage this AI-ready data set to develop a generalizable multitask AI model capable of real-time understanding of clinically significant surgical behaviors in prospectively collected real-world surgical videos. The study team programmatically queried open surgery procedures on YouTube and manually annotated selected videos to create the AI-ready data set used to train a multitask AI model for 2 proof-of-concept studies, one generating surgical signatures that define the patterns of a given procedure and the other identifying kinematics of hand motion that correlate with surgeon skill level and experience. The Annotated Videos of Open Surgery (AVOS) data set includes 1997 videos from 23 open-surgical procedure types uploaded to YouTube from 50 countries over the last 15 years. Prospectively recorded surgical videos were collected from a single tertiary care academic medical center. Deidentified videos were recorded of surgeons performing open surgical procedures and analyzed for correlation with surgical training. The multitask AI model was trained on the AI-ready video data set and then retrospectively applied to the prospectively collected video data set. Analysis of open surgical videos in near real-time, performance on AI-ready and prospectively collected videos, and quantification of surgeon skill. Using the AI-ready data set, the study team developed a multitask AI model capable of real-time understanding of surgical behaviors-the building blocks of procedural flow and surgeon skill-across space and time. Through principal component analysis, a single compound skill feature was identified, composed of a linear combination of kinematic hand attributes. This feature was a significant discriminator between experienced surgeons and surgical trainees across 101 prospectively collected surgical videos of 14 operators. For each unit increase in the compound feature value, the odds of the operator being an experienced surgeon were 3.6 times higher (95% CI, 1.67-7.62; P = .001). In this observational study, the AVOS-trained model was applied to analyze prospectively collected open surgical videos and identify kinematic descriptors of surgical skill related to efficiency of hand motion. The ability to provide AI-deduced insights into surgical structure and skill is valuable in optimizing surgical skill acquisition and ultimately improving surgical care.